Towards a true spherical camera

We present the design of a spherical imaging system with the following properties: (i) A 4π field of view that enables it to “see” in all directions; (ii) a single center of projection to avoid parallax within the field of view; and (iii) a uniform spatial and angular resolution in order to achieve a uniform sampling of the field of view. Our design consists of a spherical (ball) lens encased within a spherical detector shell. The detector shell has a uniform distribution of sensing elements, but with free space between neighboring elements, thereby making the detector partly transparent to light. We determine the optimal dimensions of the sensing elements and the diameter of the detector shell that produce the most compact point spread function. The image captured with such a camera has minimal blur and can be deblurred using spherical deconvolution. Current solid state technologies do not permit the fabrication of a high resolution spherical detector array. Therefore, in order to verify our design, we have built a prototype spherical camera with a single sensing element, which can scan a spherical image one pixel at a time. 1. WHAT IS A TRUE SPHERICAL CAMERA? Wide angle imaging has had a long history of over a century. Starting with the lens array system constructed by Scheimpflug in 1900,1 numerous designs have been proposed to capture wide fields of view. Current approaches to wide angle imaging can be classified as either dioptric (the use of refractive optics) or catadioptric (the use of refractive and reflective optics). Dioptric systems include ones that use camera clusters2–4 and fisheye lenses,6, 7 while catadioptric systems often use curved mirrors in conjunction with lenses.8–10, 12, 13 While some of these designs allow us to image the complete spherical field of view, we are yet to see a true spherical camera. A true spherical camera may be defined as one that has the following three properties: • Property A: 4π field of view. The camera should be able to “see” in all directions. In practice, it may be forced to have a small blind spot to accommodate the electronics and wiring needed to read out the image. However, we require the optics of the system itself to be able to capture the complete spherical field of view. • Property B: Single center of projection (COP). Each sensing element (e.g., pixel) of the camera will receive a bundle of light rays that can be represented by a principal ray. All the principal rays, which correspond to the viewing directions of the camera, are required to intersect at a single point, namely, the center of projection. This constraint allows for the reconstruction of perspective views from the captured spherical image. This, in turn, allows for captured images to be processed by a variety of existing computer vision algorithms that assume perspective projection. • Property C: Uniform resolution. The camera should ensure uniform sampling of the spherical FOV. Moreover, the solid angle imaged by each sensing element should be equal. In this paper, we present a spherical imaging system that satisfies all of the above properties. Our design consists of a spherical (ball) lens encased within a spherical detector shell. The detector shell has a uniform distribution of sensing elements, but with free space between neighboring elements. To achieve uniform spatial resolution, the elements are placed on a discrete spherical grid obtained by recursive subdivision of a regular polyhedron. The free space between the elements makes the detector shell partly transparent to light. As a result, the set of light rays that make it through one side of the detector shell, and are received by the spherical lens, are focused onto sensing elements on the opposite side of the detector shell. Unlike an ideal thin lens, a spherical lens is unable to focus incoming parallel rays onto a single point (see Figure 1a). We determine the optimal diameter of the detector shell and the dimensions of sensing elements that result in a camera with a compact point spread function (PSF). The image captured with such a camera has minimal blur and can be deblurred using spherical deconvolution. Another important feature to be noticed is that, the system has a large entrance aperture. For instance, a camera with a ball lens of diameter 3 inches has an entrance pupil of approximately half the diameter of Keynote Paper Human Vision and Electronic Imaging XIV, edited by Bernice E. Rogowitz, Thrasyvoulos N. Pappas, Proc. of SPIE-IS&T Electronic Imaging, SPIE Vol. 7240, 724002 © 2009 SPIE-IS&T · CCC code: 0277-786X/09/$18 · doi: 10.1117/12.817149 SPIE-IS&T/ Vol. 7240 724002-1 Downloaded from SPIE Digital Library on 16 Feb 2010 to 128.59.23.46. Terms of Use: http://spiedl.org/terms the ball lens. Consequently, the camera has a low working f-number, optics is fast and the captured images have good signal-to-noise ratio even in low-light conditions. Currently, fabrication of spherical detector shells with high pixel resolution poses technical challenges and is an active research area.27 Recent breakthroughs with respect to fabrication of sensors and electronics on curved surfaces 26, 28 suggest that the construction of spherical detectors will be possible in a few years. Meanwhile, to demonstrate the feasibility, we have built a spherical camera with a single pixel. By controlling the position and orientation of the pixel using a robotic arm, we show how a high quality, true spherical image can be captured one pixel at a time. 2. PREVIOUS WORK While many of the existing wide angle imaging systems satisfy one or more of the properties mentioned in the previous section, to our knowledge, none of these systems satisfy all. Camera clusters contain multiple cameras looking in different directions that collectively image the 4π FOV (Property A).3, 7 The images captured by the individual cameras are combined to get a spherical image mosaic. However, due to the fact that each camera has its own COP and that they cannot be co-located owing to the finite size of the cameras, camera clusters, as a whole, cannot have a single COP (Property B). This results in parallax within the FOV, which can manifest as ghosting artifacts in the final stitched image. Wide angle catadioptric systems have been extensively studied over the past couple of decades. These systems usually involve a camera viewing a curved mirror that reflects a hemispherical FOV. A number of designs have been proposed to have a single COP.8, 11 Some of these designs even allow the arrangement of two back-to-back systems so that their COPs coincide. Such an arrangement would satisfy both Properties A and B. However, these designs do not have a uniform resolution. On the other hand, some catadioptric systems have been designed to have constant resolution but not a single COP.16, 17 A spherical camera with of these systems placed back-to-back would satisfy Properties A and C, but not Property B. Finally, the mirrors used in omnidirectional cameras usually have a large curvature and tend to introduce significant aberrations like astigmatism and coma,14 thereby affecting the resolution of the captured image. One design found in nature that lends itself toward a spherical camera is the compound eye that is present in many insects. The compound eye is a collection of thousands of more-or-less spherically arranged, microscopic photoreception units called ommatidia.15 Each ommatidium has a micro lens and a receptor; and provides exactly one picture element to the brain. The individual “pixels” are then combined by the brain to form an image. Cole 18 has proposed an imaging system based on the compound eye. Here, instead of a micro lens, each ommatidium has an optic fiber which receives light from the environment. It is straightforward to extend his system into a spherical camera. The optic fibers can be arranged spherically so as to uniformly sample the spherical FOV. However, such an arrangement suffers from poor light gathering power due to the small entrance pupil of the optic fibers. 3. IMAGING WITH A TRANSPARENT SPHERE Spherical (ball) lenses with homogeneous refractive index are known to produce significant spherical aberrations. Figure 1a shows the refractive properties and the aberrations thereof of a spherical lens of diameter D = 10cm and whose refractive index μ = 1.495. The lens is unable to focus a set of incoming parallel rays onto a single focal point, but rather produces a caustic. Notice that the off-axis rays are bent through too great an angle to come into focus as the on-axis rays. Therefore, spherical lenses are seldom used in the optics of a camera. (Spherical lenses are usually used in pairs for signal coupling between optic fibers, emitters and detectors, where the effects of aberrations are not critical.) Now consider a small sensor (e,g., a pixel) placed at the effective focal length (EFL) of the spherical lens. EFL = μD/4(μ− 1) is the distance from the center of the lens at which most of the incoming parallel rays seem to converge. Figure 1b shows a circular sensor of diameter 130μm placed at the effective focus point. The sensor receives light from the entire hemisphere part of it through the lens (rays shown in the Figure 1b) and part of it directly from the surrounding environment (rays not shown in Figure 1b). We define the distribution of light flux over all incoming angles that is incident on the sensor as the angular point spread function (APSF). APSF is analogous to PSF, which is often measured as the response of an imaging system to a point source. While PSF is measure on the sensor side, APSF is its equivalent on the object side. The APSF for the aforementioned setup is computed using ray tracing and is shown on the right side of Figure 1b. The brightness detected by the sensor will correspond to a scene point along its viewing direction in a spherical environment whose appearance is convolved with the APSF. SPIE-IS&T/ Vol. 7240 724002-2 Downloaded from SPIE Digital Library on 16 Feb 2010 to 128.59.23.46. Terms of Use: http://spiedl.org/terms C 3 3 C 3 C 3 C-3 C-3 C--3 C--3 U U U (a) Aberrations in a spherical lens (μ = 1.495). 90 60 30 0 30 60 90 0 0.01 0.02 degrees Sensor Angular PSF (b) A sensor of size 130μm placed at the EFL of a spherical lens of size 10 cm. 15 10 5 0 5 10 15 0 0.01 0.02 Detector unit